feat(eap): Add a column to store arrays in EAP #7493
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This PR has a migration; here is the generated SQL for -- start migrations
-- forward migration events_analytics_platform : 0050_add_attributes_array_column
Local op: ALTER TABLE eap_items_1_local ADD COLUMN IF NOT EXISTS attributes_array JSON(max_dynamic_paths=128) CODEC (ZSTD(1)) AFTER attributes_float_39;
Distributed op: ALTER TABLE eap_items_1_dist ADD COLUMN IF NOT EXISTS attributes_array JSON(max_dynamic_paths=128) CODEC (ZSTD(1)) AFTER attributes_float_39;
Local op: ALTER TABLE eap_items_1_downsample_8_local ADD COLUMN IF NOT EXISTS attributes_array JSON(max_dynamic_paths=128) CODEC (ZSTD(1)) AFTER attributes_float_39;
Distributed op: ALTER TABLE eap_items_1_downsample_8_dist ADD COLUMN IF NOT EXISTS attributes_array JSON(max_dynamic_paths=128) CODEC (ZSTD(1)) AFTER attributes_float_39;
Local op: ALTER TABLE eap_items_1_downsample_64_local ADD COLUMN IF NOT EXISTS attributes_array JSON(max_dynamic_paths=128) CODEC (ZSTD(1)) AFTER attributes_float_39;
Distributed op: ALTER TABLE eap_items_1_downsample_64_dist ADD COLUMN IF NOT EXISTS attributes_array JSON(max_dynamic_paths=128) CODEC (ZSTD(1)) AFTER attributes_float_39;
Local op: ALTER TABLE eap_items_1_downsample_512_local ADD COLUMN IF NOT EXISTS attributes_array JSON(max_dynamic_paths=128) CODEC (ZSTD(1)) AFTER attributes_float_39;
Distributed op: ALTER TABLE eap_items_1_downsample_512_dist ADD COLUMN IF NOT EXISTS attributes_array JSON(max_dynamic_paths=128) CODEC (ZSTD(1)) AFTER attributes_float_39;
-- end forward migration events_analytics_platform : 0050_add_attributes_array_column
-- backward migration events_analytics_platform : 0050_add_attributes_array_column
Distributed op: ALTER TABLE eap_items_1_dist DROP COLUMN IF EXISTS attributes_array;
Local op: ALTER TABLE eap_items_1_local DROP COLUMN IF EXISTS attributes_array;
Distributed op: ALTER TABLE eap_items_1_downsample_8_dist DROP COLUMN IF EXISTS attributes_array;
Local op: ALTER TABLE eap_items_1_downsample_8_local DROP COLUMN IF EXISTS attributes_array;
Distributed op: ALTER TABLE eap_items_1_downsample_64_dist DROP COLUMN IF EXISTS attributes_array;
Local op: ALTER TABLE eap_items_1_downsample_64_local DROP COLUMN IF EXISTS attributes_array;
Distributed op: ALTER TABLE eap_items_1_downsample_512_dist DROP COLUMN IF EXISTS attributes_array;
Local op: ALTER TABLE eap_items_1_downsample_512_local DROP COLUMN IF EXISTS attributes_array;
-- end backward migration events_analytics_platform : 0050_add_attributes_array_column |
❌ 1 Tests Failed:
View the top 1 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
volokluev
reviewed
Nov 4, 2025
| Column( | ||
| "attributes_array", | ||
| JSON( | ||
| max_dynamic_paths=4096, |
Member
There was a problem hiding this comment.
this number is way too high. this would create an additional 4096 columns on each table, almost 14000 columns in total. and I'm fairly certain we will reach it. All it would take is 4096 differently named values across our entire customer base.
Member
There was a problem hiding this comment.
I would recommend putting it quite low. like 32 and have the shared structure take care of the rest
Contributor
Author
There was a problem hiding this comment.
Default is 1024, ClickHouse is OK having a higher number of columns. I'm not sure we need to keep a very small number like 32.
We want to take advantage of the shared structure but we'll need to upgrade to 25.8 first. At least, we should keep the default value.
volokluev
approved these changes
Nov 6, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We want to start storing array values in EAP, which could have multiple types (arrays of ints, arrays of floats, etc). This would use the new JSON column type in ClickHouse to help us store arrays of various types.